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ABSTRACT: Innovation and knowledge creation call for high-level epistemic agency and design¬ 
mode thinking, two competencies beyond the traditional scopes of schooling. In this paper, we 
discuss the need for learning analytics to support these two competencies, and more broadly, 
the demand for education for innovation. We ground these arguments on a distinctive 
Knowledge Building pedagogy that treats education as a knowledge-creation enterprise. By 
critiquing current learning analytics for their focus on static-state knowledge and skills, we argue 
for agency-driven, choice-based analytics more attuned to higher order competencies in 
innovation. We further describe ongoing learning analytics initiatives that attend to these 
elements of design. Prospects and challenges are discussed, as well as broader issues regarding 
analytics for higher order competencies. 
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1 INTRODUCTION 

... in order to bring education into line with the needs of society, it would be necessary to 
undertake a complete revision of the methods and aims of education, rather than continue to be 
satisfied with simple appeals to common sense. (Piaget, 1972, p. 16) 

Innovation is key to sustainable economic growth and solutions to complex problems in knowledge or 
innovation-driven societies (OECD, 2004). What can schools, from kindergarten to tertiary level, do to 
increase a society's capacity for innovation? This is one central question that motivates numerous "21 st 
century skills" initiatives worldwide (e.g., Binkley et al., 2012). Current education systems, as criticized 
by some innovators and experts, tend to keep students on predetermined paths to master given 
knowledge and skills instead of fostering serendipity, risk-taking, choice-making, failure, and long 
stretches of work (National Academy of Engineering, 2015); to meet increasing demands for innovation, 
K-12 education needs to "create a pedagogy, class, framework, or method where students learn from 
their mistakes without being penalized" and to "encourage creative ideas even if there is no short-term 
return or fruition of the idea" (pp. 51-52). In essence, in order to nurture creative talents, alternative 
education paradigms are needed to bring education into closer alignment with innovative practices. 
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While "education for innovation" is gaining increasing attention, learning analytics — a nascent field 
aspiring to optimise learning and teaching by discovering actionable knowledge from educational data 
(Long & Siemens, 2011) — has yet to invest much in it. This situation is not surprising, since assessment 
regimes wherein many learning analytics projects are developed stress content standards and education 
accountability, with the acquisition of static facts and routine skills treated as the main, if not sole, 
learning outcome (Schwartz & Arena, 2013). Higher order competencies essential to the dynamic and 
adaptive aspects of innovation — such as curiosity, resilience, and "way-finding" in complex spaces 
(Deakin Crick, Huang, Ahmed Shafi, & Goldspink, 2015; Dweck, 2006; Lawless, Mills, & Brown, 2002) — 
are sidelined in formal education and thus also in learning analytics projects. Despite the popular 
argument that people need to have the basics in order to innovate, learning analytics 1 that directly deal 
with high-order competencies conducive to innovation are urgently needed. 

In this paper, we argue that learning analytics should contribute to the challenge of fostering education 
for innovation in knowledge societies. To this end, we first highlight design-mode thinking driven by 
high-level epistemic agency as two central aspects of innovation. Drawing on decades of work on 
Knowledge Building (Scardamalia & Bereiter, 2003, 2014) — a distinctive educational approach to 
support education for innovation — we discuss design principles of analytics for innovation and 
knowledge creation. We present current analytics initiatives aiming to facilitate design-mode thinking 
and epistemic agency in the international Knowledge Building research community and conclude by 
discussing challenges and opportunities to advance this line of work. 

2 EDUCATION FOR INNOVATION 

Whilst my approaches to teaching and conveying information were, at times, certainly creative, 
the actual activities designed for the children, and the mode in which they were instructed to 
approach them, tended to be linear and prescriptive ... None of the children saw imagination or 
creativity as ... a significant aspect of learning. —A teacher (Claxton, Edwards, & Scale- 
Constantinou, 2006, p. 60) 

Current designs of learning analytics are mostly rooted in the dominant practices of education and 
assessment established on the basis of predefined learning objectives that focus on student acquisition 
of well-established knowledge and skills (Schwartz & Arena, 2013). The objectives are addressed in 
teaching through pre-sequenced learning contents and activities and pre-set performance measures to 
keep students on track and hold teachers accountable. A foundation underpinning this dominant 
approach to education and assessment is Bloom's (1956) Taxonomy, which classifies cognitive objectives 
into six levels that include, from low to high, Knowledge, Comprehension, Application, Analysis, 
Synthesis, and Evaluation. It has played a crucial role in expanding educational objectives beyond 
itemized subject-matter knowledge to include "intellectual abilities and skills" represented by the higher 
levels of the taxonomy (Krathwohl, 2002). Since it was first developed as an assessment framework, the 


1 We treat "learning analytics" as singular when it refers to the scholarly field or the systematic approach of mining insights 
from learning data and plural when it means specific analytical tools or applications. 
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taxonomy has been applied broadly in the development of curriculum standards, lesson plans, and 
assessment tools. 

Despite its profound influence on educators, however, Bloom's Taxonomy is less compatible with 
emergent cultures of learning (Thomas & Brown, 2011), as well as real-world knowledge practices 
deemed important for education more recently (Bereiter, 2002). One caveat of the taxonomy is that it 
treats knowledge as a static entity, or in Bloom's words, as "something filed or stored in the mind" 
(Bloom, 1956, p. 29). It places Knowledge under cognitive skills (i.e., Comprehension, Application, etc.) — 
an improper treatment a revised taxonomy tried to fix by establishing an independent Knowledge 
dimension to go along with the Cognitive Process dimension (Krathwohl, 2002). Unfortunately, in neither 
version of the taxonomy does knowledge even come close to being treated as "a means of production" 
— a more essential role that fits knowledge-based economies, where knowledge becomes objectified in 
human creations and further cognitive work would add value to it (Bereiter & Scardamalia, 1998). 

To develop new models of education that support innovation, we need to approach knowledge practices 
in line with the ways in which real-world knowledge-creating organizations operate, where knowledge is 
treated as shared conceptual artifacts or objects continually improved by members (Bereiter, 2002). 
Knowledge goals and processes cannot be pre-scripted by the central leader(s) but continually deepen 
and evolve through members' interactive input. As research shows, members of productive teams 
engage in distributed reasoning in which they perform cognitive operations (e.g., induction, deduction) 
and pass the results on to peers, who then use the results as the input for further cognitive operations 
to create new scientific theories and experiments (Dunbar, 1995). A series of small operations may lead 
to major, often unexpected advances. Therefore, education in line with real-world knowledge processes 
should treat learning as a matter of collaboratively developing shared knowledge objects and artifacts 
through sustained inquiry and interactions, a practice absent in typical learning experiences in schools 
emphasizing efficient coverage of static-state knowledge and skills. 

Education for innovation and knowledge creation demands new conceptions of and designs for learning 
to support students taking on high-level responsibilities in their knowledge work. A frontrunner in this 
direction is Knowledge Building (KB) pedagogy, which aims to refashion education in line with real-world 
knowledge-creating processes (Scardamalia & Bereiter, 2003). In a nutshell, KB emphasizes having 
students assume collective responsibility for sustained, creative work with ideas (Scardamalia, 2002). It is 
essentially knowledge creation in which students participate from the youngest grades, with learning as 
a by-product (Scardamalia & Bereiter, 2003). As in knowledge-creating organizations, KB classrooms put 
ideas, knowledge objects, or conceptual artifacts, in the centre, with all types of resources (including 
students, teachers, technology, authoritative sources) contributing synergistically towards the 
advancement of ideas. By working as a collective to advance their ideas, students take on high-level 
cognitive responsibilities including setting goals, planning inquiry, monitoring progress, seeking and 
using authoritative sources, and diagnosing problems. Pedagogical designs by KB teachers scaffold 
student collective responsibility by nurturing a safe sociocultural environment for discourse, reminding 
students of each other's contributions, helping to locate external sources, and so forth. Technology, 
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Figure 1: Main interface of Knowledge Forum (Version 6). Centre-right — one KF note, in which an 
idea about "how high does water vapour go" is presented. In a note, the basic unit of contribution in 
KF, users can specify the problem they want to address, use scaffolds to frame ideas, and add 
keywords to convey the essence of the note. Notes can be further linked in different ways, in KF terms 
of building on and referencing. Background — a KF view, a problem space created and designed by a 
KB community to organize ideas presented in notes conceptually. A view is a two-dimensional 
organizing background for notes. In a view, users have the freedom to place notes in any location. 
They can also add graphic structures, such as a concept map, a diagram, or a scene, to help organize 
notes in meaningful ways. With views and notes, KF provides an open, communal space for a 
community to engage in idea development. Bottom-left (front) — a rise-above note that presents a 
high-level summary of student ideas about "how clouds carry water." The packaged ideas can be 
accessed by clicking on the icon in the rise-above view. 


represented by a widely used environment named Knowledge Forum (Scardamalia & Bereiter, 2003), 
plays a significant role in archiving student ideas and sustaining community discourse beyond face-to- 
face classroom sessions. It is designed with functionalities to support various operations on ideas or 
knowledge objects. Briefly, it enables students to contribute ideas, in the form of notes, to a communal 
space organized into views; it provides epistemic scaffolds (e.g., "My theory," "I wonder") in notes to 
help students frame their contributions; it supports sophisticated knowledge processes such as 
synthesizing and abstracting for deeper principles through rise-above notes (which package multiple 
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ideas together), and flexible movement of ideas across the knowledge space (through note-copying, 
text-referencing, idea-exporting, etc.; see Figure 1). A recent review of the literature demonstrates the 
broad application of KB across grade levels (from kindergarten to tertiary), subject areas (e.g., science, 
mathematics, engineering, medical science), and cultural settings, with its distinctions from other 
constructivist approaches also explained (Chen & Hong, 2016). 

To summarize, KB does not treat innovation and knowledge creation as new "ingredients" of schooling, 
which is an approach embraced by many 21 st century skills initiatives. Rather, it puts forward an 
alternative education paradigm that directly places creation and innovation at its centre. Analytics 
designed for such an approach need to respond to new assessment needs emerging in this context. 
Below, we discuss two core aspects of KB as the design focus of analytics: design-mode thinking and 
high-level epistemic agency. 

2.1 Design-Mode Thinking for Continual Idea Improvement 

Authentic knowledge creation requires students to take on high-level responsibility and epistemic 
agency for continual idea improvement. Underpinning KB practices is a specific mode of thinking, which 
is now coming to be called design-mode thinking (Bereiter & Scardamalia, 2003). Design-mode thinking 
takes the kind of thinking professional knowledge builders (e.g., designers and researchers) do and 
extends it to other contexts. Specifically, knowledge builders engage in design-mode thinking when 
working with ill-defined or "wicked" problems (Rittel & Webber, 1973), which are open to different 
definitions and to tentative solution paths of unknown destinies. As work proceeds, the nature of the 
problem changes, so predetermined pathways will not suffice. Progress depends on pursuing promising 
ideas and redirecting work based on reflection on advances and failures. The pursuit of promising 
directions calls for a "design mode" of thinking, which is concerned with "the usefulness, adequacy, 
improvability, and developmental potential of ideas" (Bereiter & Scardamalia, 2003). In the design 
mode, sustained experimentation, refinement, and incremental build-on of ideas give rise to major, 
often unexpected advances. This design mode, in which knowledge creation operates, differs from 
school practices that function in a "belief mode" (i.e., focusing on acquiring "correct" answers or 
authoritative knowledge). For schools attuned to the knowledge age, a design-mode mindset should be 
all-pervasive in disciplinary courses (Bereiter & Scardamalia, 2003). This is not to pit two modes of 
thinking against each other, but to highlight the need to provide students with opportunities to venture 
into the design mode, which is essential for knowledge creation but largely missing in education. 

Studies of KB classrooms demonstrated the possibility and advantages of engaging students in 
increasingly deepening work with ideas reflecting design-mode thinking (Hakkarainen, 2003; Zhang, 
Scardamalia, Lamon, Messina, & Reeve, 2007). Students make productive choices and go beyond what 
they already know to search for deeper and more sophisticated explanations. They choose fruitful, 
"juicy," self-generated questions (e.g., how does light travel?); build on promising ideas conducive to 
productive directions (e.g., light bends because its speed changes); go beyond facts to search for deeper 
explanations (e.g., why are colours in rainbows always in the same order?); rise above diverse 
perspectives for more sophisticated conceptualizations (e.g., light as both rays and waves); find 
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connections between different areas of work to develop opportunistic collaboration (e.g., connecting 
inquiry about vision and lenses to understand nearsightedness); and so forth (Zhang et al., 2007; Zhang, 
Scardamalia, Reeve, & Messina, 2009). Design-mode thinking is central to their pursuit of increasingly 
deeper knowledge, an effort akin to real-world knowledge creation and dissimilar to covering itemized 
learning objectives informed by Bloom's Taxonomy (Bereiter & Scardamalia, 1998). 

2.2 Epistemic Agency 

Design-mode thinking aiming at knowledge creation places stronger emphasis on epistemic agency. 
"Epistemic agents should think of themselves as, and act as, legislating members of a realm of epistemic 
ends: they make the rules, devise the methods, and set the standards that bind them" (Elgin, 2013, p. 
135). Even though epistemic agency plays a role even in the most passive forms of learning, high-level 
epistemic agency inspires proactive engagement in one's learning processes. Successful learners take 
charge of their own learning, demonstrating a high degree of self-regulation, self-awareness, self- 
determination, and self-direction (Hacker, Dunlosky, & Graesser, 2009). In KB, design-mode thinking for 
continual idea improvement requires an even higher level of epistemic agency that goes beyond self¬ 
regulation in accomplishing teacher-given tasks (Scardamalia & Bereiter, 1991). In order to produce 
knowledge of consequences, students in KB classrooms make high-level decisions and choices normally 
left to the teacher: setting knowledge goals and deciding what they need to learn, choosing important 
problems to work on, engaging in long-term planning, assessing progress, analyzing idea connections, 
monitoring challenges, and choosing promising directions among multiple alternatives (Scardamalia, 
2002). Such high-level agency is essential to the development of adaptive expertise beyond routine 
skills: Adaptive experts find ill-defined, complex problems and make continual efforts to solve them 
progressively, during which deeper problems are formulated, leading to more advances (Bereiter & 
Scardamalia, 1993; Hatano & Inagaki, 1986). 

Innovation requires not only high-level epistemic agency, but also collective agency (Bandura, 2000) and 
collective cognitive responsibility (Scardamalia, 2002). Collective agency is critical when a group works 
together to attain a common goal: "A group's attainments are the product not only of shared knowledge 
and skills of its different members, but also of the interactive, coordinative, and synergistic dynamics of 
their transactions" (Bandura, 2000, p. 75). Aspects of collective agency, such as group efficacy, collective 
goal setting, and collaborative innovation (Gloor, 2005), are essentially emergent group-level properties 
that cannot be adequately addressed with measures derived from individuals. Yet they are 
fundamentally important for team-based innovation, which is usually mandatory given today's complex 
problems (e.g., climate change, global health issues). Recognizing collective agency is also critical for 
engendering collaborative learning, which happens at all agentic granularities including individuals, small 
groups, and communities (Stahl, 2013; Suthers & Verbert, 2013; Zhang et al., 2009). Working as a 
collaborative community, members not only contribute conceptual ideas, but also offer high-level 
metacognitive input to collective choice-making about what problems the community should work on, 
what types of contributions need to be made, by and with whom, and following what timeline. 

The importance of collective, epistemic agency for innovation casts doubt on dominant praxis within 
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learning analytics. In a traditional learning analytic scenario, the learners reside at the bottom of a 
hierarchy, being treated as "data objects" to be interpreted by "data clients" performed by teachers, 
institutions, and governmental agencies (Greller & Drachsler, 2012). For example, institutional data are 
fed into algorithms to predict student success, with resulting predictions delivered to the teacher on 
demand to trigger intervention (Arnold & Pistilli, 2012); analytics are also developed to support teacher 
decision-making, for instance, in orchestration of co-operative programming tasks (Berland, Davis, & 
Smith, 2015). Using learning analytics to orchestrate learning (e.g., Dietz-Uhler & Hum, 2013; Ruiperez- 
Valiente, Munoz-Merino, Leony, & Delgado Kloos, 2014), while being practically compatible with 
contemporary views of learning as social participation (Sfard, 1998), takes important knowledge- 
creation competencies away from students. The learning analytics community is becoming aware of this 
tension, arguing for treating both students and the institution as agents, who both enjoy situated, 
relative freedom to pursue successful learning (Subotzky & Prinsloo, 2011). Researchers caution the 
danger of treating learning analytics as a part of the broader bureaucratization of student learning and 
advocate for a "third-space" where students and the institution engage in negotiations about 
assumptions, beliefs, and identities (Prinsloo, Slade, & Galpin, 2012). Hence, student agency needs to be 
stressed in applications of learning analytics (Wise, 2014), as does their epistemic agency in knowledge 
processes. 

3 ANALYTICS FOR KNOWLEDGE BUILDING 

Learning analytics for KB needs to capture and provide feedback on the design-mode thinking of 
students, who act as epistemic agents to continually improve ideas. In the following sections, we 
develop two guiding principles for KB analytics, and then elaborate these principles through example 
tools and research projects. Specifically, we argue that analytics for KB needs to be 1) agency-driven and 
choice-based; and 2) progress-oriented, integrative of multi-level, multi-unit, and multi-timescale data 
produced in progressive KB discourse. Figure 2 presents a model of KB analytics highlighting two higher 
order competencies discussed in the previous section, together with the design principles to be 
elaborated below. 


Knowledge 

Building 


Analytics 


Epistemic Agency Design-Mode 



Choice-Based Progress-Oriented 

Multi-level, multi-units, multi-timescale 

Figure 2: A conceptual model of knowledge-building analytics. 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


145 






JOURNAL OF LEARNING ANALYTICS 


S 3LAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2016). Analytics for knowledge creation: Towards epistemic agency and design-mode thinking. Journal of Learning Analytics, 3(2), 139-163. 
http://dx.doi.Org/10.18608/jla.2016.32.7 

3.1 Agency-Driven, Choice-Based Analytics 

Current learning analytics primarily focus on the development of knowledge and skills as a result of 
completing predefined tasks and activities. Despite the importance of such work, "assessments 
organized around knowledge are too far removed from the realm of action and the future learner 
adaptations that education cares about" (Schwartz & Arena, 2013, p. 67). 

The importance of epistemic agency and design-mode thinking in KB leads us to recognize choices as a 
fresh angle to both understand and scaffold higher order competencies. Epistemic agency essentially 
means the capacity to make choices when advancing one's understanding. Epistemic agents "form, 
sustain, and revise their beliefs, methods, and standards" as they deem necessary, with epistemic 
autonomy to make choices regardless of circumstances (Elgin, 2013, p. 139). Choice-making in a 
collective, or in an epistemic community (Haas, 1992), further calls for consideration of an individual's 
personal appetites (and aversions) in relation to those of others for the sake of joint and interdependent 
interests. Choice-making thus becomes essential for a team's endeavours of solving wicked problems, 
during which the team members face multiple tentative solution paths and are collectively responsible 
for the consequences of their choices. Therefore, learning environments that encourage epistemic 
agency and design-mode thinking should provide learners with abundant epistemic choices. 

Attending to choices that reflect epistemic agency and design-mode thinking provides a fresh angle for 
devising learning analytics for high-level competencies. Choice-based assessment is a nascent idea 
(Schwartz & Arena, 2013). As rich forms of learning interactions and transactions are supported by 
digital environments, analyzing choices made in digital transactions provides new opportunities for 
understanding learning. As Schwartz and Arena (2013) argue, choice, rather than static-state knowledge, 
provides a stronger interpretative framework for learning outcomes — knowledge is an enabler, but 
choice captures much more, because making good choices does not depend solely on knowledge but 
also on a variety of intrapersonal, interpersonal, and environmental factors. In a similar vein, a new 
culture of learning calls for a transition from "knowledge stocks" — i.e., canons to be protected and 
transferred — to "knowledge flows" that are fluid and constantly changing with less attachment to 
institutional warrants (Thomas & Brown, 2011). Making choices in knowledge processes is a strong 
indicator of epistemic agency, as is the capability to decide among multiple choices in knowledge 
"flows." Compared to measuring knowledge "stocks" and cognitive skills, which are emphasized by 
Bloom's dominant Taxonomy, assessing choice-making in action is better aligned with the essence of 
competencies for knowledge creation and innovation. Analyzing learner choices as driven by epistemic 
agency would capture much more dynamic, adaptive, and complex aspects of learning in the digital age. 

If choices become the most critical "input" for learning analytics, the "output" of choice-based analytics 
should aim towards empowering reflexive choice-making by learners or knowledge builders. In scenarios 
where analytics are aligned with dominant frameworks of learning measurement, the non-student users 
of analytics usually make centralized decisions on learning, for good reasons in many cases. However, 
one significant challenge facing choice-based analytics for higher order competencies is to maintain the 
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balance between priming for better choices and scaffolding epistemic agency. While the establishment 
of reliable assessment of choice-making is critical, choice-based analytics could start from raising 
awareness of choices among learners and engaging them in reflecting upon their own choices in relation 
to those of others. In KB, defined as a collective process of advancing community knowledge through 
communal discourse (Scardamalia & Bereiter, 2003), choices driven by students' epistemic agency 
abound. Important choice-making reflecting high-level epistemic agency and design-mode thinking 
happens at least in the following three aspects, based on current literature: 

1. Choice-making among emergent ideas. The central business of KB is idea improvement, which 
is preconditioned by idea diversity, just as biodiversity is critical for ecosystems (Scardamalia, 
2002). One important aspect of epistemic agency in KB, hence, is to make choices among 
diverse, and sometimes competing ideas, and determine the most promising ones for a 
community to collectively labour on (Chen, Scardamalia, & Bereiter, 2015). 

2. Choice-making around emergent themes or higher-order conceptual structures. An idea never 
stands alone but is always surrounded by others. As KB progresses through community 
discourse, complex structures of ideas constantly emerge. They could be themes of inquiry that 
address distinctive principal problems (Zhang et al., 2007), or "rise-above" ideas that synthesize 
interconnected ideas (Scardamalia & Bereiter, 2014). To continually advance the community's 
knowledge, one important issue for students is to collectively grapple with these high-level, 
emergent knowledge structures. Collective choice-making around these structures, such as 
choosing which lines of inquiry to follow and deciding means to advance them, is critical for 
deepening knowledge building. 

3. Choice-making of discourse moves. KB as a community enterprise treats every community 
member as a legitimate contributor. As epistemic agents, students contribute to KB discourse in 
a variety of ways, including posing questions, theorizing, introducing new information, making 
synthesis, monitoring discussion, and so forth (Chuy et al., 2011), all of which are valued in a KB 
community. Choices made among these different "epistemic moves" figure at both single 
contributions and high-level knowledge structures (e.g., threads, views; see Figure 1). When 
making a contribution, decisions are made on which epistemic moves could help to address a 
particular knowledge problem; to advance a "thread" of inquiry, students need to analyze which 
epistemic moves are needed overall (Resendes, Scardamalia, Bereiter, Chen, & Halewood, 
2015). 

Choice-based KB analytics may begin with these key aspects of choice-making pertinent to collective 
knowledge advancement and gradually expand to other areas. 

3.2 Progress-Oriented Analytics 

While choices are important indicators of key competencies in KB, the productivity of such choices 
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needs to be further benchmarked by achieved knowledge progress over time, including progress of 
individual understanding as well as collective knowledge of a community. Given KB's focus on sustained 
idea development, KB analytics is naturally oriented to the progress of understanding emerging from 
interactive idea inputs from community members. Such a focus on the progress of understanding is 
common among extent learning analytics. For example, Khan Academy provides analytics of learning 
activities to assess and demonstrate learning progress (Ruiperez-Valiente et al., 2014); intelligent 
tutoring systems also model understanding based on one's interactions with the tutor (Allen, Snow, & 
McNamara, 2015; Desmarais & Baker, 2012). Existing work in this area will inform the implementation 
of analytics to assess knowledge progress in KB. 

The KB literature has suggested three important aspects of idea progress: conceptual depth, breadth, 
and the emergence of new strands of inquiry. The first two aspects, i.e., the depth and breadth of 
community knowledge, are achieved through idea (co-)elaboration and diversification in knowledge¬ 
building communities (Hong & Sullivan, 2009). Distinctive from most current progress-oriented analytics 
focusing on individuals, KB analytics needs to attend to both the individual and community levels. At the 
individual level, known proxies of knowledge progress include content and lexical measures (Chen, Ma, 
Matsuzawa, & Scardamalia, 2015) and semantic overlaps with curriculum standards (Teplovs & Fujita, 
2009). At the community level, researchers have experimented with techniques such as term clouds 
(Hong & Scardamalia, 2014) and topic modelling (Chen, 2014; Sun, Zhang, Jin, & Lyu, 2014) to assess 
community knowledge. Latent Semantic Analysis has been successfully applied to analyze text 
coherence (Foltz, Kintsch, & Landauer, 1998) and could be applied to assess idea coherence in KB. 
Additional behavioural measures extracted from activity logs could be used to augment the analysis of 
content coherence with insights into students' social interactions (e.g., Segedy, Kinnebrew, & Biswas, 
2015) to address basic but important questions such as "whether students incorporated ideas they read 
into those they wrote." New development is needed to better assess both dimensions of knowledge 
progression at multiple levels in KB. 

As for the third aspect, KB analytics needs to attend to emergent directions and strands of inquiry. In 
particular, assessment in KB is transformative, feeding forwards to potential progresses attainable from 
the current state of community knowledge. This is different from, for instance, analytics that 
recommend potentially helpful information, or hints given by intelligent tutors based on assessment of 
student knowledge. In contrast, since KB is a closer parallel to "swarm creativity," where emergent 
interactions lead to collective advances (Gloor, 2005), KB analytics needs to feed forwards to self¬ 
organization around emergent directions in a community (Bereiter & Scardamalia, 2013). This 
characteristic needs to be considered without losing sight of the epistemic agency principle, so that 
analytics do not take away agency, but rather enable reflective noticing and adaptive structuring, 
scaffold clearer communication, and assist "hunting" for promising uncertainties in the community. For 
example, Idea Thread Mapper (to be elaborated below) enables students' reflective noticing and 
structuring of emergent strands of inquiry (Chen, Zhang, & Lee, 2013; Zhang et al., 2015). More work is 
needed to strengthen this aspect of analytics. 
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To summarize, KB analytics, as it emphasizes agency, choice, and progress, embraces the transformative 
power of analytics, favouring a view treating "analytics as learning," in contrast to popular emphasis on 
analytics about, of, or for learning. Instead of treating learning analytics as a set of tools for evaluating 
learning and learners, analytics has the power to carry on cognitive actions to support KB processes, 
with cognition being distributed among community members, knowledge artefacts, and analytics. Such 
a view of analytics, albeit difficult to achieve, demands integral views of different units of social 
interactions at different time scales in KB. Semantic, temporal, social, and network analyses need to be 
integrated to identify distributed knowledge, idea progress, and "rotating leadership" in KB communities 
(e.g., Ma, Matsuzawa, Chen, & Scardamalia, 2016). 

4 EXAMPLES OF KNOWLEDGE BUILDING ANALYTICS 

One of the earliest analytics developed for Knowledge Forum, the most broadly used KB environment, 
was the Analytic Toolkit (ATK; Burtis, 1998). The toolkit provides comprehensive summary statistics on 
activities in a Knowledge Forum database at both the individual and community levels. An evolving suite 
of lightweight analytic applets are also available for learners to monitor the behavioural, lexical, 
semantic, and social aspects of discourse patterns (e.g., Teplovs, Donoahue, Scardamalia, & Philip, 
2007). External analytic tools with unique design goals and features, such as Knowledge Connection 
Analyzer (van Aalst, Mu, & Yang, 2016) and Knowledge Building Discourse Explorer (Oshima, Oshima, & 
Matsuzawa, 2012), have also been actively developed. 

To elaborate on the design principles introduced earlier, below we present recent research projects that 
use analytics to support students' epistemic agency and design-mode thinking in KB. For each project, 
we highlight the analytics tools oriented toward students' agency-driven choices and related idea 
progress, and synthesize the educational use of the analytics tools and their impact on KB. The main goal 
is not to present best practices, but to illuminate possibilities and to encourage further improvements. 

4.1 Promising Ideas Tool: Choice-Making among Emergent Ideas 

Idea improvement is the central concern of KB. While a pool of diverse ideas contributed by students 
provides the foundation for possible improvement, a key concern related to design-mode thinking and 
epistemic agency is the determination of which ideas produced by the community are promising and 
worth further efforts. The practice of judging the potentials of ideas is evident in creative processes of 
all kinds, when principled knowledge is scant and many competing ideas emerge at the same time 
(Bereiter, 2009). Such judgments have been explored in studies of creativity (Gardner, 1994; 
Wertheimer & Wertheimer, 1959) and creative expertise (Bereiter & Scardamalia, 1993). Choices made 
among ideas in these circumstances — propelled by epistemic agency and design-mode thinking — 
eliminate or delay other alternatives and greatly influence the trajectory of idea development. 

To devise analytics for choice-making among emergent ideas, implementation efforts were made to first 
capture the students' deliberate choices. An embedded facility for identifying promising ideas in 
Knowledge Forum was developed, with a goal to make the choice of promising ideas a routine in KB for 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


149 


JOURNAL OF LEARNING ANALYTICS 


S 3LAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2016). Analytics for knowledge creation: Towards epistemic agency and design-mode thinking. Journal of Learning Analytics, 3(2), 139-163. 
http://dx.doi.Org/10.18608/jla.2016.32.7 

students as young as 8 years old (Chen, Scardamalia, & Bereiter, 2015). With this facility, students could 
highlight a text snippet within a Knowledge Forum note as a promising idea (see Figure 3.a). Contextual 
data around a choice — information about who (the agent/student), where (the note and view in which 
an idea is tagged), when (timestamp), and for what — are all captured to afford later reflection by 
students. 

After students identify promising ideas, their choices are then harnessed by analytics to support 
collective deliberation oriented towards knowledge progress. Tagged ideas are aggregated, merged 
based on text overlaps (i.e., idea A and B are merged if A contains B), ranked based on the number of 
"hits," and visualized for students to process. In a network layout of the idea aggregation window (see 
Figure 3.b), similarities among ideas are calculated (using the character-based Damerau-Levenshtein 
similarity approach), and turned into an interactive visualization that enables students to explore 
connections among identified ideas (by clicking on the edges). In this layout, an edge between two ideas 
denotes a similarity between them that surpasses a given threshold, and is intended to encourage 
students to explore their relations. By clicking on the edge between two linked ideas, students could 
further inspect their connections, which could lead to syntheses and "rise-above" ideas. Cognition 
becomes distributed between analytics and students, but students' epistemic agency remains at the 
centre, as students take advantage of analytics to sift through identified promising ideas, choose the 
most promising ones, and export them to a new workspace for further inquiry (see Chen, Scardamalia, & 
Bereiter, 2015). 
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. Problem 

( My Theory I do not think that chickens evolved from dinosouars because 
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(a) A note with one promising idea highlighted from a third grade class. 
Figure 3: The Promising Ideas Tool. 
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0 O O Promising Ideas 



(b) The network layout of idea aggregation window from a graduate-level class. In this layout, 
semantic linkages among ideas are visualized. The user can review two ideas together by clicking on 
the edge between them. 

Figure 3: The Promising Ideas Tool. 

To date, no efforts have been made to assess the quality of student choices of promising ideas. The 
focus thus far has been on ensuring that students' epistemic agency is not undermined but empowered 
by analytics, which scaffold student attempts to improve their ideas. As demonstrated earlier, rather 
than trying to optimize choices based on trained optimal models, KB researchers applied simple text¬ 
mining algorithms and visualization techniques to build tools to uncover the high-order competency of 
identifying promising ideas among young learners. In two empirical studies conducted in third- and six- 
grade classes (eight-to-twelve-year old students), the Promising Ideas tool, coupled with designed 
pedagogical supports, has shown promise in improving student awareness of community ideas, 
promoting collaboration (reflected by social network analysis), facilitating community knowledge 
(demonstrated through content analysis and group comparisons), and advancing students' epistemic 
beliefs (measured by established instruments; Chen, 2016; Chen, Scardamalia, & Bereiter, 2015). 

4.2 Epistemic Discourse Moves Tool: Choosing Moves to Drive Discourse Forward 

In science, where KB has been broadly applied (Chen & Hong, 2016), epistemic moves to achieve deeper 
understanding include seeking plausible causal models, problematizing models, detecting flawed 
evidence, and building from counter-evidence (Perkins & Grotzer, 2000, p. 7). For KB in general, 
epistemic moves to build explanatory theories (Bereiter, 2002) include posing thought-provoking 
questions, theorizing, introducing new information, working with information, synthesizing and 
comparing, and supporting discussion (Chuy et al., 2011). An important task to support high-level 
epistemic agency, thus, is to expand the student repertoire of epistemic moves to promote their design- 
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mode thinking around ideas. One option for devising analytics to support this area is to capture student 
choices of different moves, and then engage students in reflecting on previous moves and choosing the 
next fruitful moves. These high-level knowledge processes are described as meta-discourse — the 
philosophical sense of "discourse about discourse" (Resendes et a I2015) — which aims to advance KB 
and in the meantime contribute to improving students' epistemic fluency (Morrison & Collins, 1996) in 
explanation building. 

To support meta-discourse by students, an Epistemic Discourse Moves tool has been developed for 
Knowledge Forum (Figure 4). The tool garners epistemic markers, known as scaffolds (e.g., "My 
theory...," "I need to understand...," "A better theory...") that are left by students when making 
contributions (see Figure 1). By aggregating scaffold use, the tool captures epistemic moves made by 
students and feeds this information back to them. As illustrated in Figure 4, the tool presents the counts 
of different scaffolds used by the community as a bar graph. New scaffold uses that appear after the 
tool is activated are shown in red. By clicking on a scaffold, students can also interactively filter notes in 
a Knowledge Forum view. Such functionalities are designed to help raise student awareness of their 
choices/actions, and to navigate the complex idea landscape in their KB work, with a goal to inform 
choice-making involving epistemic moves. 


WOO Scaffold Tracker 

Scaffold Use in View - Birds are Smart! 

Count 

0 5 10 15 20 25 


I need to understand 


Important information 
+source 


My theory 

The information is 
important because 


Figure 4: Epistemic Discourse Moves tool in a second grade view about birds. 

Simple as the tool appears to be, when it was piloted in a second-grade class (seven-to-eight-year-old 
children), students immediately realized they had contributed too many theories and questions but not 
enough information (Resendes et al., 2015). Visualizations created forms of feedback that allowed 
second-grade students to attend to facets of their knowledge work that are otherwise inaccessible and 
indigestible to them. With pedagogical interventions designed around this analytics tool, students were 
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able to incorporate into their discourse moves that are sophisticated for their age (Resendes et al., 
2015). Their choices of moves — both initial ones when contributing an idea and later ones after 
epistemic gaps are recognized — are clear indicators of epistemic agency and design-thinking efforts to 
achieve knowledge progress. The tool does not prescribe actions for students. Instead, it initiates group 
dialogues, with students maintaining the epistemic agency to choose the next moves. 

4.3 Idea Thread Mapper: Grappling with Emergent Knowledge Structures 

Besides analytics to trace choices around discourse moves, further support is needed to capture choices 
made by students around higher order, collective structures of KB in their unfolding discourse. Instead 
of following pre-scripted tasks and procedures of inquiry, students in a KB community need to enact 
high-level agency for defining what needs to be investigated and how. They co-construct the collective 
focus and landscape of the community's research and discourse, as collective structures that frame the 
shared, emergent objects of inquiry, the unfolding strands of inquiry and norms of contributions, and 
deeper needs of knowledge to be addressed. However, in current online discussions, students' ideas are 
contributed in distributed online posts that extend over time; there is a lack of tools to help students 
monitor the collective landscape, progress, and directions emerged from their distributed, long-term 
discourse (cf. Hewitt, 2001; Suthers, Vatrapu, Medina, Joseph, & Dwyer, 2008). Consequently, their 
online discourse may become disconnected and ill-grounded, and lack progressively deepening moves to 
advance understanding (Zhang et al., 2009). 

To support students' reflective monitoring and adaptation of collective structures of inquiry practices, 
Zhang and colleagues designed a timeline-based, collective discourse-structuring tool: Idea Thread 
Mapper (ITM) (Chen, Zhang, & Lee, 2013; Zhang et al., 2015). On top of micro-level representations of 
ideas using online postings and build-ons (physical conversation threads), ITM incorporates "idea 
threads" as an emergent structure in online discourse. Each idea thread, as shown in Figure 5, includes 
as a sequence of discourse entries (possibly involving several build-on trees — see Figure 1) that 
investigates a shared epistemic object (e.g., allergies), as an unfolding strand of inquiry (Zhang et al., 
2007). Features of ITM signify collective knowledge structures reflected in the online KB discourse, 
including 1) using the thread topics and "Journey of Thinking" syntheses to highlight the shared 
epistemic objects being investigated and absences of knowledge to be addressed by the community; 2) 
using timeline-based discourse mapping to visualize the unfolding, interrelated strands of inquiry 
practices focusing on the epistemic objects; and 3) retrieving members' participatory roles in the 
different strands of inquiry. The collective landscape of a whole KB initiative is mapped out as a cluster 
of idea threads that investigate a set of interrelated problems through the contributions of all members. 
Visualization tools further show the intensity of contributions in each thread and cross-thread 
connections, including cross-thread build-on links and connective contributions that simultaneously 
address two or more objects of investigation. Automated text analyses drawing upon LightSIDE 
(formerly known as TagHelper — see Rose et al., 2008) and topic modelling help students retrieve notes 
related to each object of inquiry and profile the notes based on various types of moves to deepen 
understanding (e.g., questions, explanations, facts). 
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Example analyses: Distribution of notes in threads Cross-thread connection 


Figure 5: A map of idea threads created by a Grade 5/6 classroom studying the human body. Each 
coloured stripe in the upper map represents an idea thread extending from the first until the last note 
contributed addressing its focal object of inquiry (e.g., allergies). Each square represents a note; a blue 
line between two notes represents a build-on link; a vertical dotted line marks notes that are shared 
between different threads discussing interrelated issues. The example analyses (bottom) show the 
distribution of notes in the different idea threads and conceptual connections between the threads 

based on cross-thread common notes. 
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ITM can be used by teachers to trace progress of student discourse in addressing important knowledge 
themes. More importantly, it is used by students to enact their high-level responsibility for structuring 
and deepening collective discourse over time to achieve sustained idea advancement. ITM-supported 
classroom designs engage students in reflexive monitoring of their community space and decision¬ 
making on next steps of community discourse, including existing knowledge objects to be deepened and 
new objects of inquiry to be addressed. A set of studies was conducted among third- through sixth- 
grade classrooms (eight-to-twelve-year-old students) equipped with Knowledge Forum and ITM (Chen et 
al., 2013; Zhang, Chen, Chen, & Mico, 2013; Zhang et al., 2015). In each KB initiative that extended over 
multiple months focusing on one core science area, students used ITM to co-structure their online 
discourse around significant epistemic objects as unfolding idea threads, and, on an ongoing basis, 
update the threads in reflection of new progress. The analysis showed that, with proper pedagogical 
supports, the young students were able to construct and interpret the idea thread maps and use the 
information to reflect on advances and connections in the unfolding lines of inquiry, decide on deeper 
issues to be addressed, and plan inquiry activities to address these needs. The ITM-aided reflective 
monitoring and structuring of online discourse played a positive role in increasing student awareness of 
their community's collective foci and progress and fostering more connected discourse moves to 
generate deep and coherent explanations of scientific issues (Zhang et al., 2015). 

As noted above, ITM and the reflective processes it enables support students' agency-driven choices to 
frame and direct productive lines of inquiry emerging from diverse contributions. Automated analyses in 
ITM further support students' sense-making and syntheses of collective idea progress at the community 
level. Further design and research needs to better support and understand the micro-macro interaction 
to capture how diverse individual contributions give rise to the community's shared foci and strands of 
inquiry, and how representations of the shared foci and strands of inquiry are used by individuals to 
guide their reflective choices about their ongoing participation and contribution. 

5 DISCUSSION AND CONCLUSION 

One central argument of this article is as follows: To consider learning analytics for higher order 
competencies, we need first to look beyond dominant views of educational objectives and attend to 
competencies critical for innovation-driven societies. Grounded in an extended body of literature, we 
highlight high-level epistemic agency and design-mode thinking as two pivotal components of education 
for knowledge creation. We then situate the discussion within the distinctive Knowledge Building 
pedagogy and propose that KB analytics needs to be agency-driven, choice-based, and progress- 
oriented. To demonstrate possibilities, we present ongoing learning analytics initiatives representing 
these design principles. Despite varied educational contexts and research goals, the analytics projects 
presented show promise for facilitating these higher order competencies (i.e., epistemic agency and 
design-mode thinking) in empirical studies. 

The second argument we made is that student agency should not be trivialized when devising learning 
analytics for higher order competencies. When data analytics are applied in education, special care 
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needs to be taken regarding student agency because of the complexity of power relations among 
stakeholders (Slade & Pinsloo, 2013). This is especially true if the goal of education is to help learners 
become autonomous human beings (Kamii, 1984) and become (or not cease to be) creators and 
innovators (Scardamalia & Bereiter, 2014). This argument builds upon the caution that learning analytics 
should be explicit about its underlying beliefs in epistemology, assessment, and pedagogies (Knight, 
Buckingham Shum, & Littleton, 2014), and also embraces a call for attention to student agency and 
sense-making dialogues when analytics are applied (Wise, 2014). Our emphasis on epistemic agency and 
choices deepens the idea of aiding and capturing student choice-making to assess learning (Schwartz & 
Arena, 2013). In the examples outlined in this article, analytics describe non-evaluative information, 
while students evaluate the presented information and make choices oriented towards progress. The 
relationship between analytics and learners becomes more collaborative and dynamic, with cognition 
distributed on both sides. In this way, learning analytics is no longer confined within the acquisition of 
knowledge or skills, but becomes enlarged to cover cultivation of metacognitive facts, epistemic fluency, 
and habits of mind (Bereiter & Scardamalia, 2013) — all representing higher order competencies. 

Our arguments are destined to be criticized regarding issues of validity and reliability in assessment 
development. For instance, the use of choice as an interpretative framework of learning is still new and 
needs substantial work to establish validity and reliability. Nonetheless, "the methodological demand of 
reliability coincides with a tendency of people to take an essentialist perspective that reifies 
assessments into stable traits or essences of a person — individual properties that do not change" 
(Schwartz and Arena, 2013, p. 117). The issues of validity and reliability need to be revisited when 
constructing adaptive and dynamic assessment of competencies. Building analytics to assess design¬ 
mode thinking is an underexplored and difficult task. One fundamental difficulty is a lack of 
understanding about what constitutes design-mode thinking, as the underlying constructs and what 
features it consistently reveals in innovative processes are understudied. An early fixation with validity 
and reliability can suffocate innovation in choice-based analytics. An alternative path, which may help 
mitigate this difficulty, is to rethink whether accurate measurement aspired by educational assessment 
(cf. Ellis, 2013) would be absolutely necessary for learning analytics in general. While learning analytics is 
predominantly tied to assessment, if the goal is to advance designs of analytics, avoiding obsession with 
accurate assessment would enable iterative designs in the absence of principled knowledge (Bereiter, 
2009). This approach presents a transition from the popular approach of working backwards from 
standards when devising analytics, towards an approach emphasizing feeding/orworc/s to the discovery 
of students' higher order competencies. 

Learning analytics focusing on choice-making is in an early stage and faces serious challenges. In addition 
to validity and reliability, more work is clearly needed to conceptualize choices made in learning 
processes. Research of self-regulated learning deals with student choices of learning tactics (Winne, 
1995; Winne & Hadwin, 2013). However, important work is needed to extend research into choices 
beyond individual learning to collaborative KB and knowledge creation, as is basic research of human 
choices (e.g., Figner et a I., 2010; Shafir, 1993). Another more practical challenge facing choice-based 
learning analytics is the difficulty of capturing choices made in broader contexts. A more holistic picture 
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of choice-making in complex environments requires collection of multimodal data beyond online digital 
traces (e.g., Olguin, Gloor, & Pentland, 2009). 

Future work is clearly needed to address these challenges faced by analytics for KB and knowledge 
creation. Research programs are forming to tackle these issues in the KB research community. The 
adopted approach, again, is to instantiate conditions and environments for higher level competencies, 
instead of working backwards from education standards. This approach opens the door to exceed 
targeted educational outcomes and to uncover new competencies beyond the consensual expectations 
of stakeholders (Scardamalia, Bransford, Kozma, & Quellmalz, 2012). Learning analytics is well- 
positioned to democratize learning accomplishments by moving away from a traditional fixation with 
standards and tests to designing environments, analytics, and practices that nurture learners' creative 
moves or "knowledge flows" (Thomas & Brown, 2011). 
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